Video tracking with deep siamese networks and bayesian optimization

ABSTRACT

An apparatus, method, system and computer readable medium for video tracking. An exemplar crop is selected to be tracked in an initial frame of a video. Bayesian optimization is applied with each subsequent frame of the video by building a surrogate model of an objective function using Gaussian Process Regression (GPR) based on similarity scores of candidate crops collected from a search space in a current frame of the video. A next candidate crop in the search space is determined using an acquisition function. The next candidate crop is compared to the exemplar crop using a Siamese neural network. Comparisons of new candidate crops to the exemplar crop are made using the Siamese neural network until the exemplar crop has been found in the current frame. The new candidate crops are selected based on an updated surrogate model.

TECHNICAL FIELD

Embodiments generally relate to computer vision. In particular,embodiments relate to video tracking. More particularly, embodimentsrelate to efficient video tracking using deep Siamese networks andBayesian optimization.

BACKGROUND

Early video tracking approaches include feature-based approaches andtemplate matching algorithms that attempt to track specific features ofan object or even the object as a whole. Feature-based approaches uselocal features, including points and edges, keypoints, SIFT(Scale-Invariant Feature Transform), HOG (Histogram of OrientedGradients) features and deformable parts. Conversely, template-basedmatching takes the object as a whole, offering the potential advantageof treating complex templates or patterns that cannot be modeled bylocal features alone.

Recently, deep learning models have been applied to video tracking toleverage the benefits of learning complex functions from large datasets. While deep learning models offer the potential of improvedrobustness for tracking, they present two significant challenges tovideo tracking. First, many deep learning tracking models are too slowfor practical use because they require online training. Second, manydeep learning tracking models, when trained offline, are based onclassification approaches, so that they are limited to class-specificsearches and frequently require the aggregation of many image patches,and thus many passes through the network, in order to locate the object.In light of these two difficulties, several contemporarystate-of-the-art deep learning-based tracking models have been developedas generic object trackers in an effort to obviate the need for onlinetraining and also to improve the generalization of the tracker. A recentstate-of-the-art model applies a regression-based approach to train ageneric tracker, GOTURN, offline to learn a generic relationship betweenappearance and motion. Several deep learning techniques additionallyincorporate motion and occlusion models, including particle filteringand optical flow.

Even with the recent successes in video tracking, there is still a voidin state-of-the-art video tracking workflows that fully integrate deeplearning techniques with classical statistics and machine learningapproaches. Most state-of-the-art video trackers lack, for instance, acapacity to generate systematic belief states (e.g., through expliciterror and uncertainty measures), or ways to seamlessly incorporatecontextual and scene structure, or to adaptively encode temporalinformation (e.g., by imposing intelligent search stopping conditionsand bounds) and the ability to otherwise directly and inferentiallycontrol region proposal generation or sampling methods in a precise andprincipled way.

BRIEF DESCRIPTION OF THE DRAWINGS

The various advantages of the embodiments will become apparent to oneskilled in the art by reading the following specification and appendedclaims, and by referencing the following drawings, in which:

FIG. 1A is a diagram illustrating a simplified process for videotracking an object in a video clip according to an embodiment;

FIG. 1B is a simplified flow diagram of a method of tracking an objectin a video clip according to an embodiment;

FIG. 2A is a flow diagram of a method of training a Siamese networkaccording to an embodiment;

FIG. 2B is a simplified block diagram for determining similarity betweentwo objects according to an embodiment;

FIG. 3A is a flow diagram of a method of incorporating Bayesianoptimization with deep learning to track an object in a video accordingto an embodiment;

FIG. 3B is a graph illustrating {circumflex over (f)}(x, t) for DOPaccording to an embodiment;

FIG. 4 is a flow diagram of an example method of Siamese-DynamicBayesian Tracking according to an embodiment;

FIG. 5 is a diagram illustrating IOU results per frame for testing theperformance of a Siamese-Dynamic Bayesian Tracking Model (SDBTM) againstthree baseline video tracking models;

FIG. 6 is an exemplary block diagram of a computer system efficientvideo tracking with deep Siamese networks and Bayesian optimizationaccording to an embodiment;

FIG. 7 is an illustration of an example of a semiconductor packageapparatus according to an embodiment;

FIG. 8 is a block diagram of an exemplary processor according to anembodiment; and

FIG. 9 is a block diagram of an exemplary computing system according toan embodiment.

In the following detailed description, reference is made to theaccompanying drawings which form a part hereof wherein like numeralsdesignate like parts throughout, and in which is shown by way ofillustration embodiments that may be practiced. It is to be understoodthat other embodiments may be utilized, and structural or logicalchanges may be made without departing from the scope of the presentdisclosure. Therefore, the following detailed description is not to betaken in a limiting sense, and the scope of embodiments is defined bythe appended claims and their equivalents. In an all in all in all.

DESCRIPTION OF EMBODIMENTS

Technology to provide a statistically-principled approach to objecttracking that integrates a deep learning neural network model with adynamic Bayesian optimization framework to enable intelligent computervision systems capable of high-level vision tasks such as, for example,scene and behavior understanding. The robustness of deep learningcombined with the efficiency of Bayesian optimization produces anaccurate tracker that requires a minimal number of object queries.Together, the combination generates confidence and uncertainty measuresthat guide a localization process by balancing prior knowledge with newquery information.

The video tracking problem is treated as a dynamic (i.e., temporallyevolving) optimization problem. Embodiments employ a deep Siamesenetwork to provide a similarity function with respect to a referenceobject to be tracked. A dynamic Bayesian optimization framework isapplied to efficiently locate the object in subsequent frames of thevideo. The Bayesian optimization framework encodes spatio-temporalinformation for efficient object tracking in video. Temporal coherenceis enforced in the tracking output through the use of a dynamic,temporally-evolving, objective function. Using Gaussian Process priors,the dynamic objective function is modeled, representing the location ofa tracked object in each frame of the video. By exploiting temporalcorrelations, the search space of a video frame is queried in astatistically principled and efficient way to minimize the number oftimes an image is pushed through the Siamese network. This isaccomplished by using an acquisition function, defined from the modeleddynamic objective function, to decide where to sample next in the searchspace of the video frame.

Various operations may be described as multiple discrete actions oroperations in turn, in a manner that is most helpful in understandingthe claimed subject matter. However, the order of description should notbe construed as to imply that these operations are necessarily orderdependent. In particular, these operations may not be performed in theorder of presentation. Operations described may be performed in adifferent order than the described embodiment. Various additionaloperations may be performed and/or described operations may be omittedin additional embodiments.

References in the specification to “one embodiment,” “an embodiment,”“an illustrative embodiment,” etc., indicate that the embodimentdescribed may include a particular feature, structure, orcharacteristic, but every embodiment may or may not necessarily includethat particular feature, structure, or characteristic. Moreover, suchphrases are not necessarily referring to the same embodiment. Further,when a particular feature, structure, or characteristic is described inconnection with an embodiment, it is submitted that it is within theknowledge of one skilled in the art to affect such feature, structure,or characteristic in connection with other embodiments whether or notexplicitly described. Additionally, it should be appreciated that itemsincluded in a list in the form of “at least one of A, B, and C” can mean(A); (B); (C); (A and B); (B and C); (A and C); or (A, B, and C).Similarly, items listed in the form of “at least one of A, B, or C” canmean (A); (B); (C); (A and B); (B and C); (A and C); or (A, B, and C).

The disclosed embodiments may be implemented, in some cases, inhardware, firmware, software, or any combination thereof. The disclosedembodiments may also be implemented as instructions carried by or storedon one or more transitory or non-transitory machine-readable (e.g.,computer-readable) storage medium, which may be read and executed by oneor more processors. A machine-readable storage medium may be embodied asany storage device, mechanism, or other physical structure for storingor transmitting information in a form readable by a machine (e.g., avolatile or non-volatile memory, a media disc, or other media device).As used herein, the term “logic” and “module” may refer to, be part of,or include an application specific integrated circuit (ASIC), anelectronic circuit, a processor (shared, dedicated, or group), and/ormemory (shared, dedicated, or group) that execute one or more softwareor firmware programs having machine instructions (generated from anassembler and/or a compiler), a combinational logic circuit, and/orother suitable components that provide the described functionality.

In the drawings, some structural or method features may be shown inspecific arrangements and/or orderings. However, it should beappreciated that such specific arrangements and/or orderings may not berequired. Rather, in some embodiments, such features may be arranged ina different manner and/or order than shown in the illustrative figures.Additionally, the inclusion of a structural or method feature in aparticular figure is not meant to imply that such feature is required inall embodiments and, in some embodiments, it may not be included or maybe combined with other features.

Video tracking is used in a wide variety of applications in computervision, such as, video compression, video surveillance, vision-basedcontrol, human computer interfaces, medical imaging, augmented reality,and robotics. Video tracking enables an exemplar object identified in aninitial frame of a video clip to be automatically located and tracked ineach successive frame of the video clip. A video clip may consist of atleast two or more frames, and in some cases may include one hundred ormore frames. The exemplar object is identified in the initial frameusing a bounding box. The bounding box is a rectangle that is drawnaround the exemplar object at a true center coordinate x in the initialframe of the video clip. The exemplar object, also referred to as anexemplar crop, is tracked in subsequent frames by identifying acandidate crop thought to be the equivalent of the exemplar crop.Tracking in subsequent frames is measured using an IOU (Intersection ofUnion) metric. The IOU metric consists of a number between zero (0) andone (1). If the IOU in a subsequent frame is 0, then there is no overlapmeasured between the candidate crop and the exemplar crop. If the IOU ina subsequent frame is 1, then the overlap measured between the candidatecrop and the exemplar crop is perfect, meaning that the exemplar objecthas been accurately identified in the subsequent frame.

FIG. 1A is a diagram illustrating video tracking of an object in a videoclip 100 according to an embodiment. For simplistic reasons, the diagramonly shows two subsequent frames of the video clip 100. The video clip100 shows a first subsequent video frame 102 followed by a secondsubsequent video frame 104. Video frames 102 and 104 are in sequence.Video frames 102 and 104 include a plurality of objects with an exemplarobject 106 identified as having a white bounding box (i.e., the truebounding box) drawn around a pedestrian 108 as an indication that thepedestrian 108 is the object being tracked. Although not shown in FIG.1A, in an initial frame of the video clip 100, a black bounding box wasdrawn around the pedestrian 108 to indicate the pedestrian 108 as theexemplar object to be tracked throughout each frame of the video clip.The black bounding box drawn around the pedestrian in the initial framehas a true center coordinate x. In the subsequent video frame 102 ofFIG. 1A, a candidate crop 110 is identified using a black bounding box112. The black bounding box 112 partially overlaps with the right sideof pedestrian 108 and is therefore several pixels to the left of centerx of the true bounding box. Thus, the candidate crop 110 would result inan IOU metric somewhere between 0 and 0.25. In the subsequent videoframe 104 that occurs immediately after frame 102 in the video clip 100,a candidate crop 114 is identified using a black bounding box 116. Theblack bounding box 116 partially overlaps with the upper left side ofpedestrian 108 and is therefore several pixels to the right of center xof the true bounding box. Thus, the candidate crop 114 would result inan IOU metric somewhere between 0 and 0.2.

In embodiments, multiple candidate crops may be sampled in eachsubsequent frame in order to find a candidate crop that yields thehighest similarity to the exemplar crop. This is accomplished using aSiamese neural network. A sample candidate crop is passed through theSiamese network to obtain a similarity score. Bayesian optimization isused to efficiently sample a space of candidate crops where the exemplarobject may be located based on prior similarity scores. For example, ina frame in which the selected candidate crops are identified as havingsimilarity scores between 0.5 and 0.8, the candidate crop with asimilarity score of 0.8 will be chosen for that frame as identifying thelocation of the exemplar object. The next frame will build upon previousframe locations of where the exemplar object is located to determinewhere to look next in sampling candidate crops for that frame. In otherwords, embodiments utilize temporal coherence when tracking an objectfrom one frame to the next. This process repeats for each frame of thevideo clip 100.

FIG. 1B is a simplified flow diagram of an exemplary method 120 forvideo tracking an object in a video clip according to an embodiment. Themethod 120 may generally be implemented in a computer system such as,for example, the computer system 600 as described below with referenceto FIG. 6. More particularly, the method 120 may be implemented in oneor more modules as a set of logic instructions stored in a machine-orcomputer-readable storage medium such as random access memory (RAM),read only memory (ROM), programmable ROM (PROM), firmware, flash memory,etc., in configurable logic such as, for example, programmable logicarrays (PLAs), field programmable gate arrays (FPGAs), complexprogrammable logic devices (CPLDs), and fixed-functionality logichardware using circuit technology such as, for example, applicationspecific integrated circuit (ASIC), complementary metal oxidesemiconductor (CMOS) or transistor-transistor logic (TTL) technology, orany combination thereof.

For example, computer program code to carry out operations shown in themethod 120 may be written in any combination of one or more programminglanguages, including an object-oriented programming language such asJAVA, SMALLTALK, C++ or the like and conventional procedural programminglanguages, such as the “C” programming language or similar programminglanguages. Additionally, logic instructions might include assemblerinstruction, instruction set architecture (ISA) instructions, machineinstruction, machine depended instruction, microcode, state settingdata, configuration data for integrated circuitry, state informationthat personalizes electronic circuitry and/or other structuralcomponents that are native to hardware (e.g., host processor, centralprocessing unit (CPU), microcontroller, etc.). The process begins inblock 122, where it immediately proceeds to block 124.

In block 124, an object to be tracked is selected from an initial framezero of video clip. The selected object is referred to as an exemplarcrop. The process then proceeds to block 126.

In block 126, a bounding box is placed over the exemplar crop at centercoordinate x. The process then proceeds to block 128.

In block 128, the video clip is advanced to the next frame. The processthen proceeds to block 130.

In block 130, an object is identified in the current frame as the objectto be tracked. The object is referred to as a candidate crop. Theprocess then proceeds to decision block 132.

In decision block 132, it is determined whether the candidate crop isthe same as the exemplar crop. If it is determined that the candidatecrop is not the same as the exemplar crop, the process proceeds back toblock 130 to determine a new candidate crop for the current frame.

Returning to decision block 132, if it is determined that the candidatecrop is the same as the exemplar crop, the process proceeds to block134. In block 134, the object is identified as being tracked and thetracking data is saved for the current frame. The process then proceedsto decision block 136.

In decision block 136, it is determined whether the current frame is theend of the video clip. If the current frame is not the end of the videoclip, the process then proceeds back to block 128 where the video clipis advanced to the next frame.

Returning to decision block 136, if the current frame is the end of thevideo clip, the process proceeds to block 138, where the trackingprocess ends.

The above described process may seem easy but locating the object to betracked in each successive frame can be expensive as well astime-consuming. To provide a more efficient process for video tracking,embodiments integrate a dynamic Bayesian optimization framework inconjunction with deep learning for video object tracking. A deep Siameseneural network is used to provide a similarity function with respect toa reference object to be tracked. A dynamic Bayesian optimizationprocedure (i.e., a temporally-evolving Gaussian process) is then appliedto efficiently locate the object in subsequent frames.

Siamese neural networks were chosen for several reasons. First, they cangeneralize beyond what they were trained on, thus, enabling them todifferentiate between classes they were not exposed to during training.And because they can generalize to classes they were not trained on;Siamese networks do not need to be retrained every time they arerequired to differentiate between something new. Instead of beingtrained to recognize what an image is, Siamese networks learn asimilarity function that allows them to differentiate images. They donot perform classification, and therefore require very little training.In fact, they are easy to train.

FIG. 2A is a flow diagram illustrating an exemplary method 200 fortraining a Siamese network according to an embodiment. The method 200may generally be implemented in a computer system such as, for example,the computer system 600 as described below with reference to FIG. 6.More particularly, the method 200 may be implemented in one or moremodules as a set of logic instructions stored in a machine- orcomputer-readable storage medium such as random access memory (RAM),read only memory (ROM), programmable ROM (PROM), firmware, flash memory,etc., in configurable logic such as, for example, programmable logicarrays (PLAs), field-programmable gate arrays (FPGAs), complexprogrammable logic devices (CPLDs), and fixed-functionality logichardware using circuit technology such as, for example, applicationspecific integrated circuit (ASIC), complementary metal oxidesemiconductor (CMOS) or transistor-transistor logic (TTL) technology, orany combination thereof.

For example, computer program code to carry out operations shown in themethod 200 may be written in any combination of one or more programminglanguages, including an object-oriented programming language such asJAVA, SMALLTALK, C++ or the like and conventional procedural programminglanguages, such as the “C” programming language or similar programminglanguages. Additionally, logic instructions might include assemblerinstruction, instruction set architecture (ISA) instructions, machineinstruction, machine depended instruction, microcode, state settingdata, configuration data for integrated circuitry, state informationthat personalizes electronic circuitry and/or other structuralcomponents that are native to hardware (e.g., host processor, centralprocessing unit (CPU), microcontroller, etc.).

A Siamese network architecture is composed of two identical networks,with each network coupled to a different input to compare the similaritybetween an exemplar image Z and a candidate image X. The trainingobjective is for the network to learn a general similarity function, f(z, x), for object tracking that compares an exemplar crop Z to acandidate crop X and returns a similarity score. If the similarity scoreis a high score, the two images depict the same object. If thesimilarity score is a low score, the two images depict differentobjects. The process begins in block 202 where the process immediatelyproceeds to block 204.

In block 204, the Siamese network applies an identical transformation 1to both exemplar image Z and candidate image X. The transformationallows for the inputs to be represented in a manner that is useful forthe task of determining whether the objects are similar or not. Theprocess then proceeds to block 206.

In block 206, the transformed representations of the two input imagesare then combined using a function g that is trained to learn thegeneral similarity function on deep convolutional network features, suchthat f (z, x)=g (φ(z), φ(x)). The process then proceeds to block 208.

In block 208, the network is trained on positive and negative pairsusing a logistic loss function l (y, v)=log (1+exp (−yv)), where v isthe real-valued score of an exemplar-candidate pair and yϵ{−1, +1} isits ground truth label. In other words, y indicates whether thecandidate image X corresponds with the same object as the exemplar imageZ. When the network is given a candidate image X that is similar to theexemplar image Z, y should yield a +1. When the network is given acandidate image X that is different from exemplar image Z, y shouldyield a −1.

The loss function l (y, v) is chosen to encapsulate the task that thenetwork is to learn. When the loss is small, the network is doing a goodjob on carrying out the desired task. When the loss is big, the networkis not doing a good job on carrying out the desired task. Thus, when theoverall loss function is minimized, a good similarity function isachieved.

The positive and negative pairs use images of the same object asexemplar image Z and images X of objects that are different fromexemplar image Z. For example, if exemplar image Z is an image of aperson, then positive pairs would use images of that person and negativepairs might use images of a different person or a different object.Positive image exemplars were defined as a function of the center offsetdistance from the ground truth and a network stride length. Image sizeswere normalized for consistency during training. The process thenproceeds to block 210.

In block 210, during training of the Siamese network, network parametersare calculated and updated by applying Stochastic Gradient Descent to:

arg min_(θ) E _((z,x,y))[l(y,f(z,x;θ))],

where E_((z,x,y)) represents a statistical average that is computed overthe data distribution, and the parameters of the Siamese network, suchas, for example, the weights and the biases of the network, aresummarized by the variable 0. This finds parameter values of the networkthat minimize the expected loss. As previously indicated, the loss isdefined by the task; given an exemplar Z and a candidate X, determinewhether the candidate X is of the same object found in exemplar Z. Inother words, the training process finds the correct weights for thenetwork that minimize the difference in the loss between y, which is thetrue value, and f, which is the value predicted by the network. Theprocess then proceeds to decision block 212.

In decision block 212, it is determined whether a good similarityfunction has been achieved by the Siamese network. A good similarityfunction is achieved when the overall loss function is minimized. If itis determined that a good similarity function has not been achieved,then the process proceeds back to block 208, where the Siamese networkcontinues to be trained and the parameters (θ) of the network continueto be calculated and updated (in block 210).

Returning to block 212, if it is determined that a good similarityfunction has been achieved, the process proceeds to block 214, where thefinal parameters obtained from the training can be implemented in theSiamese convolutional network.

FIG. 2B is a simplified block diagram 220 for determining similaritybetween two objects according to an embodiment. Diagram 220 shows afirst input 222 and a second input 224. The first input 222 comprisesthe exemplar image Z or reference image. The second input 224 comprisesthe candidate image X. In one embodiment, exemplar image Z and candidateimage X are both 255×255×3 in size, indicating that the images aresquare images with red, green and blue color channels. First and secondinputs 222 and 224 are each passed to networks 226 and 228,respectively, where they are transformed into an internalrepresentation, shown as output tensors 230 and 232, that is useful forthe task of determining whether the input images are similar or not.Thus, the dimensions of the output tensors 230 and 232 are 22×22×128,which has a smaller height and length, but and a larger depth. Theoutput tensors 230 and 232 are convolved at 234 to generate a similarityscore output 236. Similarity scores for a batch of sample images arelater rendered in a 20×20×1 search grid, shown as 238, using a Gaussianprocess.

The architecture of a Siamese network is very simple. It consists ofsymmetric twin networks that are joined together by a conjoining layerat the top of the network. In one embodiment, the twin networks may eachuse a five-layer Cony-Net (convolutional network) architecture, withpooling layers after the first and second layers, and stride lengths oftwo and one throughout. It is well known to one skilled in the relevantart(s) that cony-nets are typically used in Siamese networks when thedata involves images.

As previously indicated above, embodiments use the Siamese network as adeep learning approach to video tracking by receiving crops of an imagewhere an object may be located in a frame and scoring that crop with anexemplar crop to see if there's a good match. One could do an exhaustivesearch of each frame in the video by taking all of the crops from aframe and passing each crop through the Siamese network to obtainsimilarity scores for each crop in comparison to the exemplar crop, andwhichever crop yields the highest similarity score could be confidentlychosen as the location of the object being tracked. This would obviouslybe very inefficient as it would require millions and millions of cropsto be passed through the Siamese network for tracking an objectthroughout every frame of the video. Passing every crop from an entireframe through the Siamese network is computationally expensive as wellas time-consuming. Thus, frames could not be processed in real time dueto the enormous amount of crops in each frame.

In order to be very efficient with regards to sampling a search space tolocate where the object to be tracked could potentially be in thesubsequent frames of the video, embodiments use a technique calledDynamic Bayesian Optimization. Bayesian optimization finds the maximumvalue of an unknown function using statistical principles. The reasonit's called Dynamic Bayesian Optimization in the instant case is becausethe function to be optimized is a location of the object to be trackedin the video from one frame to the next, which changes temporally. Inother words, the location of the object changes over time.

Object tracking in video is defined as a dynamic optimization problem(DOP):

DOP={max f(x,s.t.×ϵF(t)⊂S,tϵT}

where:

-   -   Sϵ        ^(D), with S in the search space;    -   f: S×T→        is the temporally-evolving objective function which yields a        maximum when the input x matches the ground-truth of the target        object; F(t) is the set of all feasible solutions xϵF(t)⊂S at        time t.

FIG. 3A is a flow diagram of a method 300 of incorporating Bayesianoptimization with deep learning to track an object in a video accordingto an embodiment. The method 300 may generally be implemented in acomputer system such as, for example, the computer system 600 asdescribed below with reference to FIG. 6. More particularly, the method300 may be implemented in one or more modules as a set of logicinstructions stored in a machine- or computer-readable storage mediumsuch as RAM, ROM, PROM, firmware, flash memory, etc., in configurablelogic such as, for example, PLAs, FPGAs, CPLDs, and fixed-functionalitylogic hardware using circuit technology such as, for example, ASIC, CMOSor TTL technology, or any combination thereof.

For example, computer program code to carry out operations shown in themethod 300 may be written in any combination of one or more programminglanguages, including an object-oriented programming language such asJAVA, SMALLTALK, C++ or the like and conventional procedural programminglanguages, such as the “C” programming language or similar programminglanguages. Additionally, logic instructions might include assemblerinstruction, instruction set architecture (ISA) instructions, machineinstruction, machine depended instruction, microcode, state settingdata, configuration data for integrated circuitry, state informationthat personalizes electronic circuitry and/or other structuralcomponents that are native to hardware (e.g., host processor, centralprocessing unit (CPU), microcontroller, etc.).

The process begins in block 302, where the process immediately proceedsto block 304. In block 304, a surrogate model is built to learn a latentobjective function from collected samples of candidate crops from aframe in a video. Similarity scores for each sample are obtained bypassing the candidate crops in sequence through the Siamese network toobtain several different similarity scores for the frame. The similarityscores are used to draw a graph (i.e., surrogate model) over the imageto obtain a best guess estimate as to what the similarity scores wouldbe in places where candidate crops were not extracted and passed throughthe Siamese network. The graph drawn over the image formulates hills andvalleys indicative of the similarity scores for a frame, wherein thehigher the similarity score, the more promising a candidate crop at thatlocation may be the true exemplar object to be tracked. Gaussian ProcessRegression (GPR) is used to render the surrogate model. The uncertaintyin the surrogate is quantified using GPR. The process then proceeds toblock 306.

In block 306 plausible points to sample from the objective function aredetermined in the search space based on the surrogate model. The processthen proceeds to block 308.

In block 308, a secondary optimization of a surrogate dependentacquisition function a(x, t) is used to decide where to obtain the nextsample in the frame. The acquisition function strikes a balance betweenexploring new regions in the search space and exploiting informationobtained from previous samples of the objective function. A novelacquisition function, memory-score expected-improvement (MS-EI), wasderived as follows:

MS-EI=(μ(x))−f(x*)−ξ)Φ(Z)+σ(x)ρ(Z)

where:

-   -   Z=(μ(x)−f(x*)−ξ)/σ(x);    -   x*=argmax f(x);    -   Φ and ρ denote the PDF (Probability Distribution Function) and        CDF (Cumulative Distribution Function) of the standard normal        distribution, respectively; and    -   ξ=(α·mean[f(x)]_(D)·n^(q))⁻¹        α and q are tunable parameters that depend on the scale of the        objective function. In one embodiment, α=1, q=1.1. D denotes the        sample data set, and n is the sample iteration number, with        |D|=n; mean[f(x)]_(D) is the sample mean of the previously        observed values ξ serves to balance the exploration-exploitation        trade-off to the specificity of a particular search. Thus, MS-EI        employs a cooling schedule so that exploration is encouraged        early in the search; however, the degree of exploration is        conversely dynamically attenuated for exploitation as the search        generates sample points with larger output values. μ(x) is the        expected score of a particular crop x without passing the crop        through the Siamese network. σ(x) encodes the uncertainty of the        particular prediction for μ(x).

The acquisition function is used to decide where to look next to obtaina sample crop to track the exemplar object in the frame. In other words,what's the most promising location to look in the image to find theexemplar object. The acquisition function is a much cheaper function toevaluate than using the Siamese network. The acquisition function isapplied to an entire search space. This can be done instantaneously, andis therefore, a cheap function to apply vs. passing each crop throughthe Siamese network. It is applied over all possible crops in the searchspace of the frame, and provides a best guess approximation as to whatthe Siamese network would indicate as a true similarity score if thecrops actually passed through the Siamese network. The crop with thehighest acquisition result is selected as the next sample. That crop isextracted from its location and passed through the Siamese network toobtain a new similarity score. The graph is then updated with the newsimilarity score from the Siamese network for the extracted crop.

In one embodiment, the process of finding a new sample using theacquisition function, obtaining a crop based on the new sample, andpassing the crop through the Siamese network to obtain a similarityscore is repeated N times and the highest similarity score is chosen asthe location of the object to be tracked in that frame. The process thenproceeds back to block 304 to find the exemplar object in the next frameof the video.

In another embodiment, the process of finding a new sample using theacquisition function, obtaining a crop based on the new sample, andpassing the crop through the Siamese network to obtain a similarityscore is repeated until a confidence level is sufficiently high that thecrop for the current location is the best guess of where the exemplarobject is located in the frame. The process then proceeds back to block304 to find the exemplar object in the next frame of the video.

In one embodiment, the search space may be a 20×20 image grid to allowthe process to execute in real-time or close to real-time. In otherembodiments, the user may choose a different search space based on oneor more of the demands of the problem being solved, the resolution ofthe images and the hardware used.

Using the acquisition function to obtain a best guess approximation ofwhat the Siamese network would indicate as the true similarity scoreminimizes the number of times needed to push an image crop through theSiamese network while efficiently sampling where the object could belocated in a frame for video tracking in a timely manner.

A Gaussian process (GP) defines a prior distribution over functions witha joint NORMALITY assumption. {circumflex over (f)} is denoted as therealization of the Gaussian process: {circumflex over (f)}˜GP(μ, K). GPis fully specified by the mean μ: X→

and covariance K:X×X→

, K((x, t),(x′, t′))=E[({circumflex over (f)}(x, t)−μ(x, t))({circumflexover (f)}(x′, t′)−μ(x′, t′))], where K(⋅, ⋅)≤1 and X=S×T. μ is theaverage, the best guess as to what the Siamese network would score thatregion or that crop if it was actually passed through the Siamesenetwork. K, the covariance or kernel function, indicates similarity ofnearby points. For example if one point in the search space gets a scoreof 0.8, then the neighboring points should have scores close to 0.8.Thus, K enforces a continuity for the Gaussian process.

DOP f(x, t) is modeled as a spatio-temporal GP where the objectivefunction at time t represents a slice off constrained at t. The dynamicGP model encapsulates statistical correlations in space and time.Furthermore, the GP may enable tracking the location of an objectexpressed as the temporally-evolving maximum of the objective functionf(x, t).

In one embodiment, {circumflex over (f)}(x, t)˜GP(0, K({x, t}, {x′, t′})), where (x, t)ϵ

³ (x is the bounding-box spatial location), and K is the covariancefunction of the zero-mean spatio-temporal GP. For simplicity, we assumethat K is both stationary and separable of the form:

K({circumflex over (f)}(x,t),{circumflex over (f)}(x′,t′))=K_(S)(x,x′)·K _(T)(t,t′)

where K_(S) and K_(T) are the spatio and temporal covariance functions,respectively. In one embodiment, Matern kernel functions are used inexperiments and to train the spatial and temporal covariance functionsindependently, following separable assumptions. Matern kernel functionsare well known to one skilled in the relevant art(s).

FIG. 3B is a diagram 320 illustrating {circumflex over (f)}(x, t) forDOP according to an embodiment. Diagram 320 is an example of the graphdrawn at blocks 304 and 308 of FIG. 3A above in determining where toobtain the next sample. Diagram 320 illustrates the dynamic Gaussianprocess to video tracking over time. The vertical axis representsfunction {circumflex over (f)}(x, t) while the horizontal axisrepresents time. Slicing function {circumflex over (f)}(x, t) along aline perpendicular to the time axis represents {circumflex over (f)}(x,t) for a particular frame. When determining the next sample, one wouldlike for {circumflex over (f)}(x, t) to be as large as possible overtime. In tracking an object, one chooses the optimal location of theobject not only with respect to the current frame object, but also withrespect to all of the information gathered from previous frames. Thisenables one to predict what will happen in future frames as well.

References 302 illustrates data from the current frame in which thesystem is trying to find the exemplar object and draw a bounding boxaround that object. Reference 301 illustrates data from all of theprevious frames which may be useful in locating the exemplar object inthe current frame. Reference 303 illustrates estimated data for futureframes based on all of the data from the current frame 302 and previousframes 301.

A Siamese-Dynamic Bayesian Tracking Model (SDBTM) utilizes the deepSiamese cony-net, previously described above, as a general objectsimilarity function in combination with a Dynamic Bayesian Optimization(BO) framework to encode spatio-temporal information for efficientobject tracking in video. The deep Siamese cony-net is trained to obtainthe general object similarity function. The target object to be trackedis selected in the initial frame, at t=0 of the video, where the videocontains T total frames. The target object is identified as beinglocated within a ground truth bounding box centered at x. The GPRapproximation is rendered over a resized search grid of size d×d, where,in one embodiment, d=20 for computational efficiency. One skilled in therelevant art(s) would know that the value of d is not limited to 20. Inother embodiments, other values for d may be used that also providecomputational efficiency. Subsequently, upscaling is applied over theoriginal search space dimensions. To allow for changes in the scale ofthe target object, each evaluation of an image crop is rendered by theSiamese network as a triplet score, where the similarity score iscomputed for the current crop compared to the exemplar crop at threescales: {1.00−p, 1.00, 1.00+p}, where p is heuristically set at 0.05.

FIG. 4 is a flow diagram of an example method 400 of the Siamese-DynamicBayesian Tracking Method (SDBTM) according to an embodiment. The method400 may generally be implemented in a system such as, for example, thecomputer system 600 as described in FIG. 6. More particularly, themethod 400 may be implemented in one or more modules as a set of logicinstructions stored in a machine- or computer-readable storage mediumsuch as RAM, ROM, PROM, firmware, flash memory, etc., in configurablelogic such as, for example, PLAs, FPGAs, CPLDs, and fixed-functionalitylogic hardware using circuit technology such as, for example, ASIC, CMOSor TTL technology, or any combination thereof.

For example, computer program code to carry out operations shown in themethod 400 may be written in any combination of one or more programminglanguages, including an object-oriented programming language such asJAVA, SMALLTALK, C++ or the like and conventional procedural programminglanguages, such as the “C” programming language or similar programminglanguages. Additionally, logic instructions might include assemblerinstruction, instruction set architecture (ISA) instructions, machineinstruction, machine depended instruction, microcode, state settingdata, configuration data for integrated circuitry, state informationthat personalizes electronic circuitry and/or other structuralcomponents that are native to hardware (e.g., host processor, centralprocessing unit (CPU), microcontroller, etc.).

The process begins in block 402, where the process immediately proceedsto block 404. In block 404, a dynamic Gaussian Process (GP) model istrained. The hyper-parameters of the GP model are tuned using a smallportion of the training data. This procedure makes certain that thehyper-parameters, which define the spatial scale of the model, aregenerally correct. For example, the GP has a parameter called scalefactor. If the scale factor is set incorrectly, then the GP will makepredictions that don't match well with the image scale. If the imagesare large, then the GP might be too refined or vice versa.

Once the dynamic GP model has been trained, each frame of the video mustbe analyzed to determine the location of the exemplar object Z to betracked. The process then proceeds to block 406.

In block 406, for each frame of the video, the most likely location ofthe exemplar object Z must be found. We begin this process by using theacquisition function to indicate which candidate crop to select as thenext sample. In other words, the acquisition function takes all of thepixels in a d×d search space and provides an approximate score for everylocation in the search space using acquisition function MS-EI. Thehigher the acquisition score, the more promising the location. Thus, thelocation with the highest acquisition score is selected as the nextsample. The process then proceeds to block 408.

In block 408, a candidate crop X from the location with the highestacquisition score is passed through the Siamese network to obtain asimilarity score f(x, t) indicating how similar the candidate crop X isto the exemplar crop Z. Again, a very high similarity score indicates agood match while a very low similarity score indicates a poor match. Theprocess then proceeds to block 410.

In block 410, the coordinates and similarity scores for the candidatecrop scored by the Siamese network are added to the dataset. The processthen proceeds to block 412.

In block 412, the GPR approximation is rendered over the d×d searchgrid. Using a data set of crop centers x and f(x, t), the GP returns amean μ(x) for the coordinate x and a standard deviation σ(x) for allpoints in the d×d grid. The acquisition function is used to computeMS-EI(x), which scores each coordinate in the d×d grid according to howpromising it is to look at this point as the next sample. The higher theMS-EI score, the more promising the point. The process then proceeds toblock 414.

In block 414, the d×d grid data is upsampled to the dimensions of searchspace S, where S is the version of the original search space. In oneembodiment, S is the size of the original image. Thus, the d×d grid isapproximated over the entire image. The process then proceeds to block416.

In block 416, f(x*) is updated in the MS-EI equation. f(x*) representsthe highest score assigned any crop by the Siamese network in thecurrent frame. The process then proceeds to decision step 418.

In decision step 418, it is determined whether the maximum number ofiterations per frame is reached. If the maximum number of iterationshave been reached, the location of the exemplar crop Z in the currentframe is represented the crop assigned the highest score by the Siamesenetwork in the current frame. The maximum number of iterations per frameis pre-determined. The process then proceeds to decision block 420.

In decision block 420, it is determined whether there are additionalframes in the video. If it is determined that there are additionalframes in the video, the video advances to the next frame and theprocess returns to block 408 to begin determining the location of theexemplar crop Z in a new frame.

Returning to decision block 420, if it is determined that there are nomore frames in the video, the process proceeds to block 422, where theprocess ends.

Returning to decision block 418, if it is determined that the maximumiterations per frame have not been reached, the process returns to block406 to continue determining the location of the exemplar crop Z in thecurrent frame.

FIG. 5 is a diagram 500 illustrating IOU results per frame for testingthe performance of SDBTM against three baseline video tracking models.The three baseline video tracking models include template matching usingnormalized cross correlation (TM), the MOSSE tracker algorithm and ADNET(2017, CVPR), a state of the art, deep reinforcement learning-basedvideo tracking algorithm. Each model was tested using the “CFNET” videotracking data set. The test video included 200 frames. As shown indiagram 500, the vertical axis corresponds to the IOU and the horizontalaxis corresponds to the frame number of the video.

As shown in diagram 500, each of the comparison models suffered fromeither significant long-term tracking deterioration or episodicinstability. For example, the MOSSE tracker fails to track after frame30. The TM model fails to track from approximately frame 23 to frame 85,and the ADNET model fails to track after frame 170. The SDBTM used 80samples per frame (cf. region proposal systems commonly rely onthousands of image queries). The SDBTM model, for the most part,exhibited general stability in tracking the 200 frames and showed strongquantitative tracking result, with a mean IOU of 0.56 and a standarddeviation (std) IOU of 0.17. Table 1 shows the experimental resultssummary for the mean IOU and the standard deviation IOU for each model.The SDBTM model exhibited the best results summary with a mean IOU of0.56 and a standard deviation IOU of 0.17. ADNET came in second placewith a mean IOU of 0.47 and a standard deviation IOU of 0.23. MOSSE andTM showed lower mean IOU's and larger or approximately equivalentstandard deviation IOU's.

TABLE 1 TM MOSSE ADNET SDBTM Mean IOU 0.26 0.10 0.47 0.56 Std IOU 0.220.25 0.23 0.17

FIG. 6 is an exemplary block diagram of a computer system 600 forefficient video tracking with deep Siamese networks and BayesianOptimization according to an embodiment. The computer system 600includes multiprocessors such as a first processor 602 (e.g., hostprocessor, central processing unit/CPU) and a second processor 604(e.g., graphics processing unit/GPU). The first processor or CPU 602 isthe central or main processor for carrying out instructions of computerprograms, such as, for example, a method for efficient video tracking.The second processor or GPU 604 is primarily used to render 2-D and 3-Dgraphics, such as, for example, rendering the surrogate model using GPR.The GPU 604 may also be utilized to assist the CPU 602 in non-graphicscomputations. The CPU 602 and/or the GPU 604 may include a core regionwith one or more processor cores (not shown).

The computer system 600 also includes multiple compute engines toprovide artificial machine intelligence. The compute engines include aneuromorphic compute engine 606 and a DSP (Digital signal processor)608. The neuromorphic compute engine 606 is a hardware based acceleratorused to increase the performance of deep neural networks. Theneuromorphic compute engine 606 may be used to run neural networks, suchas, for example, the Siamese neural network used to compare thesimilarity between an exemplar image/crop Z and a candidate image/cropX, as described above. The DSP 608 is an on-chip hardware block designedto run deep neural networks at high speed and low power withoutcompromising accuracy. The DSP 608 may be used to accelerate deeplearning inferences at the edge. Thus, the DSP 608 may be used formachine learning to train a Siamese network a similarity function usedto differentiate between two objects/images and to provide a similarityscore between two input images/crops in real-time.

The CPU 602, GPU 604, and the compute engines 606 and 608 arecommunicatively coupled to an integrated memory controller (IMC) 610.The IMC 610 is coupled to a system memory 612 (volatile memory,3-D)(Point memory). The CPU 602, GPU 604, and the compute engines 606and 608 may also be coupled to an input/output (I/O) module 616 thatcommunicates with mass storage 618 (e.g., non-volatile memory/NVM, harddisk drive/HDD, optical disk, solid state disk/SSD, flash memory), oneor more neural compute sticks (NCS) 624, such as, for example, theIntel® Movidius™ NCS (a USB-based deep learning/self-contained deviceused for artificial intelligence (AI) programming at the edge), andnetwork interface circuitry 626 (e.g., network controller, networkinterface card/NIC).

The one or more NCS(s) 624 may provide dedicated deep neural networkcapabilities to the multiprocessors (602 and 604) and the computeengines (606 and 608) at the edge. Each of the one or more NCS(s) 624include a VPU (Vision Processing Unit) to run real-time deep neuralnetworks directly from the device to deliver dedicated high-performanceprocessing in a small form factor. In embodiments, the one or moreNCS(s) 624 may be used to perform a similarity function that yields asimilarity score between an exemplar object/crop Z and a candidateobject/crop X.

The network interface circuitry 626 may provide off platformcommunication functionality for a wide variety of purposes, such as, forexample, cellular telephone (e.g., Wideband Code Division MultipleAccess/W-CDMA (Universal Mobile Telecommunications System/UMTS),CDMA2000 (IS-856/IS-2000), etc.), Wi-Fi (wireless Fidelity, e.g.,Institute of Electrical and Electronics Engineers/IEEE 802.11-2007,Wireless Local Area Network/LAN Medium Access Control (MAC) and physicallayer (PHY) specifications, 4G LTE (Fourth Generation Long TermEvolution), Bluetooth, WiMAX (e.g., IEEE 802.16-2004, LAN/MAN BroadbandWireless LANS), Global Positioning System (GPS), spread spectrum (e.g.,900 MHz), and other radio frequency (RF) telephony purposes. Otherstandards and/or technologies may also be implemented in the networkinterface circuitry 626. In one embodiment, the network interfacecircuitry 626 may enable communication with various cloud services toperform AI tasks in the cloud.

Although the CPU 602, the GPU 604, the compute engines 606 and 608, theIMC 610, and the I/O controller 616 are illustrated as separate blocks,these components may be implemented as a system on chip (SOC) 628 on thesame semiconductor die.

The system memory 612 and/or the mass memory 618 may be memory devicesthat store instructions 614, which when executed by the processors 602and/or 604 or the compute engines 606 and/or 608, cause the computersystem 600 to perform one or more aspects of methods 120, 200, 300 and400 for efficient video tracking with deep Siamese networks and Bayesianoptimization, described above with reference to FIGS. 1B, 2A, 3A and 4.Thus, execution of the instructions 614 may cause the computer system600 to integrate deep learning with Bayesian optimization to performefficient video tracking.

FIG. 7 shows a semiconductor package apparatus 700 (e.g., chip) thatincludes a substrate 702 (e.g., silicon, sapphire, gallium arsenide) andlogic 704 (e.g., transistor array and other integrated circuit/ICcomponents) coupled to the substrate 702. The logic 704, which may beimplemented in configurable logic and/or fixed-functionality logichardware, may generally implement one or more aspects of the methods120, 200, 300 and 400 (FIGS. 1B, 2A, 3A and 4), already discussed.

FIG. 8 illustrates a processor core 800 according to one embodiment. Theprocessor core 800 may be the core for any type of processor, such as amicro-processor, an embedded processor, a digital signal processor(DSP), a network processor, or other device to execute code. Althoughonly one processor core 800 is illustrated in FIG. 8, a processingelement may alternatively include more than one of the processor core800 illustrated in FIG. 8. The processor core 800 may be asingle-threaded core or, for at least one embodiment, the processor core800 may be multithreaded in that it may include more than one hardwarethread context (or “logical processor”) per core.

FIG. 8 also illustrates a memory 870 coupled to the processor core 800.The memory 870 may be any of a wide variety of memories (includingvarious layers of memory hierarchy) as are known or otherwise availableto those of skill in the art. The memory 870 may include one or morecode 805 instruction(s) to be executed by the processor core 800,wherein the code 805 may implement the methods 120, 200, 300 and 400(FIGS. 1B, 2A, 3A and 4), already discussed. The processor core 800follows a program sequence of instructions indicated by the code 805.Each instruction may enter a front end portion 810 and be processed byone or more decoders 820. The decoder 820 may generate as its output amicro operation such as a fixed width micro operation in a predefinedformat, or may generate other instructions, microinstructions, orcontrol signals which reflect the original code instruction. Theillustrated front end portion 810 also includes register renaming logic825 and scheduling logic 830, which generally allocate resources andqueue the operation corresponding to the convert instruction forexecution.

The processor core 800 is shown including execution logic 850 having aset of execution units 855-1 through 855-N. Some embodiments may includea number of execution units dedicated to specific functions or sets offunctions. Other embodiments may include only one execution unit or oneexecution unit that can perform a particular function. The illustratedexecution logic 850 performs the operations specified by codeinstructions.

After completion of execution of the operations specified by the codeinstructions, back end logic 860 retires the instructions of the code805. In one embodiment, the processor core 800 allows out of orderexecution but requires in order retirement of instructions. Retirementlogic 865 may take a variety of forms as known to those of skill in theart (e.g., re-order buffers or the like). In this manner, the processorcore 800 is transformed during execution of the code 805, at least interms of the output generated by the decoder, the hardware registers andtables utilized by the register renaming logic 825, and any registers(not shown) modified by the execution logic 850.

Although not illustrated in FIG. 8, a processing element may includeother elements on chip with the processor core 800. For example, aprocessing element may include memory control logic along with theprocessor core 800. The processing element may include I/O control logicand/or may include I/O control logic integrated with memory controllogic. The processing element may also include one or more caches.

Referring now to FIG. 9, shown is a block diagram of a computing system900 in accordance with an embodiment. Shown in FIG. 9 is amultiprocessor system 900 that includes a first processing element 970and a second processing element 980. While two processing elements 970and 980 are shown, it is to be understood that an embodiment of thesystem 900 may also include only one such processing element.

The system 900 is illustrated as a point-to-point interconnect system,wherein the first processing element 970 and the second processingelement 980 are coupled via a point-to-point interconnect 950. It shouldbe understood that any or all of the interconnects illustrated in FIG. 9may be implemented as a multi-drop bus rather than point-to-pointinterconnect.

As shown in FIG. 9, each of processing elements 970 and 980 may bemulticore processors, including first and second processor cores (i.e.,processor cores 974 a and 974 b and processor cores 984 a and 984 b).Such cores 974 a, 974 b, 984 a, 984 b may be configured to executeinstruction code in a manner similar to that discussed above inconnection with FIG. 8.

Each processing element 970, 980 may include at least one shared cache996 a, 996 b. The shared cache 996 a, 996 b may store data (e.g.,instructions) that are utilized by one or more components of theprocessor, such as the cores 974 a, 974 b and 984 a, 984 b,respectively. For example, the shared cache 996 a, 996 b may locallycache data stored in a memory 932, 934 for faster access by componentsof the processor. In one or more embodiments, the shared cache 996 a,996 b may include one or more mid-level caches, such as level 2 (L2),level 3 (L3), level 4 (L4), or other levels of cache, a last level cache(LLC), and/or combinations thereof.

While shown with only two processing elements 970, 980, it is to beunderstood that the scope of the embodiments are not so limited. Inother embodiments, one or more additional processing elements may bepresent in a given processor. Alternatively, one or more of processingelements 970, 980 may be an element other than a processor, such as anaccelerator or a field programmable gate array. For example, additionalprocessing element(s) may include additional processors(s) that are thesame as a first processor 970, additional processor(s) that areheterogeneous or asymmetric to processor a first processor 970,accelerators (such as, e.g., graphics accelerators or digital signalprocessing (DSP) units), field programmable gate arrays, or any otherprocessing element. There can be a variety of differences between theprocessing elements 970, 980 in terms of a spectrum of metrics of meritincluding architectural, micro architectural, thermal, power consumptioncharacteristics, and the like. These differences may effectivelymanifest themselves as asymmetry and heterogeneity amongst theprocessing elements 970, 980. For at least one embodiment, the variousprocessing elements 970, 980 may reside in the same die package.

The first processing element 970 may further include memory controllerlogic (MC) 972 and point-to-point (P-P) interfaces 976 and 978.Similarly, the second processing element 980 may include a MC 982 andP-P interfaces 986 and 988. As shown in FIG. 9, MC's 972 and 982 couplethe processors to respective memories, namely a memory 932 and a memory934, which may be portions of main memory locally attached to therespective processors. While the MC 972 and 982 is illustrated asintegrated into the processing elements 970, 980, for alternativeembodiments the MC logic may be discrete logic outside the processingelements 970, 980 rather than integrated therein.

The first processing element 970 and the second processing element 980may be coupled to an I/O subsystem 990 via P-P interconnects 976 986,respectively. As shown in FIG. 9, the I/O subsystem 990 includes P-Pinterfaces 994 and 998. Furthermore, I/O subsystem 990 includes aninterface 992 to couple I/O subsystem 990 with a high-performancegraphics engine 938. In one embodiment, bus 949 may be used to couplethe graphics engine 938 to the I/O subsystem 990. Alternately, apoint-to-point interconnect may couple these components.

In turn, I/O subsystem 990 may be coupled to a first bus 916 via aninterface 996. In one embodiment, the first bus 916 may be a PeripheralComponent Interconnect (PCI) bus, or a bus such as a PCI Express bus oranother third generation I/O interconnect bus, although the scope of theembodiments is not so limited.

As shown in FIG. 9, various I/O devices 914 (e.g., biometric scanners,speakers, cameras, sensors) may be coupled to the first bus 916, alongwith a bus bridge 918 which may couple the first bus 916 to a second bus920. In one embodiment, the second bus 920 may be a low pin count (LPC)bus. Various devices may be coupled to the second bus 920 including, forexample, a keyboard/mouse 912, communication device(s) 926, and a datastorage unit 919 such as a disk drive or other mass storage device whichmay include code 930, in one embodiment. The illustrated code 930 mayimplement the methods 120, 200, 300 and 400 (FIGS. 1B, 2A, 3A and 4),already discussed, and may be similar to the code 805 (FIG. 8), alreadydiscussed. Further, an audio I/O 924 may be coupled to second bus 920and a battery 910 may supply power to the computing system 900.

Note that other embodiments are contemplated. For example, instead ofthe point-to-point architecture of FIG. 9, a system may implement amulti-drop bus or another such communication topology. Also, theelements of FIG. 9 may alternatively be partitioned using more or fewerintegrated chips than shown in FIG. 9.

Additional Notes and Examples

Example 1 includes at least one computer readable medium, comprising aset of instructions, which when executed by one or more computingdevices, cause the one or more computing devices to select an exemplarcrop to be tracked in an initial frame of a video, apply Bayesianoptimization with each subsequent frame of the video to build asurrogate model of an objective function using Gaussian ProcessRegression (GPR) based on similarity scores of candidate crops collectedfrom a search space in a current frame of the video, and determine whereto proficiently obtain a next candidate crop in the search space usingan acquisition function, compare the next candidate crop to the exemplarcrop using a Siamese neural network, and continue comparison of a newcandidate crop to the exemplar crop using the Siamese neural networkuntil the exemplar crop has been found in the current frame, wherein thenew candidate crop is selected based on an updated surrogate model.

Example 2 includes the at least one computer readable medium of example1, wherein instructions to build a surrogate model of the objectivefunction further comprises instructions, which when executed by the oneor more computing devices, cause the one or more computing devices toobtain candidate crops of an image in the search space of the currentframe of the video, obtain a similarity score for each of the candidatecrops by passing each of the candidate crops through the Siamese neuralnetwork, and draw a graph over the image using each similarity scorefrom each of the candidate crops to interpolate similarity scores oflocations in the frame that have not yet been sampled.

Example 3 includes the at least one computer readable medium of example1, wherein instructions to determine where to proficiently obtain thenext candidate crop in the search space using the acquisition functionfurther comprises instructions, which when executed by the one or morecomputing devices, cause the one or more computing devices to apply theacquisition function to each of the candidate crops in the search spaceto obtain an approximation of a similarity score for the candidate cropsas if each of the candidate crops had been passed through a Siamesenetwork without passing the candidate crops through the Siamese network,and select a candidate crop from the search space that has a maximumacquisition function result as the next candidate crop.

Example 4 includes the at least one computer readable medium of example1, wherein the acquisition function comprises a memory-scoreexpected-improvement (MS-EI) acquisition function.

Example 5 includes the at least one computer readable medium of example1, wherein instructions to compare the next candidate crop to theexemplar object further comprises instructions, which when executed bythe one or more computing devices, cause the one or more computingdevices to pass the next candidate crop through the Siamese network toobtain a similarity score, update the surrogate model with the newsimilarity score, and find a maximum point on the updated surrogatemodel to use as the new candidate crop.

Example 6 includes the at least one computer readable medium of example1, wherein the surrogate model formulates a graph of hills and valleysrepresentative of a location of a tracked object in each frame of thevideo.

Example 7 includes the at least one computer readable medium of example1, wherein the Siamese neural network is trained as a general objectsimilarity function.

Example 8 includes the at least one computer readable medium of example1, wherein the search space comprises a d×d grid, and wherein thesurrogate model is rendered over the d×d grid.

Example 9 includes the at least one computer readable medium of example1, wherein instructions to continue comparison of the new candidate cropto the exemplar crop using the Siamese neural network until the exemplarcrop has been found in the current frame further comprises instructions,which when executed by the one or more computing devices, cause the oneor more computing devices to compare N new candidate crops to theexemplar crop using the Siamese neural network and select the newcandidate crop with the highest similarity score as the exemplar crop inthe frame.

Example 10 includes the at least one computer readable medium of example1, wherein instructions to continue comparison of the new candidate cropto the exemplar crop using the Siamese neural network until the exemplarcrop has been found in the current frame further comprises instructions,which when executed by the one or more computing devices, cause the oneor more computing devices to continue comparison of the new candidatecrop to the exemplar crop using the Siamese neural network until aconfidence level in the similarity score of a current new candidate cropis sufficiently high to indicate that the exemplar crop has been found.

Example 11 includes a semiconductor package apparatus comprising one ormore substrates, and logic coupled to the one or more substrates,wherein the logic includes one or more of configurable logic orfixed-functionality hardware logic, the logic coupled to the one or moresubstrates to select an exemplar crop to be tracked in an initial frameof a video, apply Bayesian optimization with each subsequent frame ofthe video to build a surrogate model of an objective function usingGaussian Process Regression (GPR) based on similarity scores ofcandidate crops collected from a search space in a current frame of thevideo, and determine where to proficiently obtain a next candidate cropin the search space using an acquisition function, compare the nextcandidate crop to the exemplar crop using a Siamese neural network, andcontinue comparison of a new candidate crop to the exemplar crop usingthe Siamese neural network until the exemplar crop has been found in thecurrent frame, wherein the new candidate crop is selected based on anupdated surrogate model.

Example 12 includes the semiconductor package apparatus of example 11,wherein logic to build a surrogate model of the objective functionfurther includes logic coupled to the substrate to obtain candidatecrops of an image in the search space of the current frame of the video,obtain a similarity score for each of the candidate crops by passingeach of the candidate crops through the Siamese neural network, and drawa graph over the image using each similarity score from each of thecandidate crops to interpolate similarity scores of locations in theframe that have not yet been sampled.

Example 13 includes the semiconductor package apparatus of example 11,wherein logic to determine where to proficiently obtain the nextcandidate crop in the search space using the acquisition functionfurther includes logic coupled to the substrate to apply the acquisitionfunction to each of the candidate crops in the search space to obtain anapproximation of a similarity score for the candidate crops as if eachof the candidate crops had been passed through a Siamese network withoutpassing the candidate crops through the Siamese network, and select acandidate crop from the search space that has a maximum acquisitionfunction result as the next candidate crop.

Example 14 includes the semiconductor package apparatus of example 11,wherein the acquisition function comprises a memory-scoreexpected-improvement (MS-EI) acquisition function.

Example 15 includes the semiconductor package apparatus of example 11,wherein logic to compare the next candidate crop to the exemplar objectfurther includes logic coupled to the substrate to pass the nextcandidate crop through the Siamese network to obtain a similarity score,update the surrogate model with the new similarity score, and find amaximum point on the updated surrogate model to use as the new candidatecrop.

Example 16 includes the semiconductor package apparatus of example 11,wherein the surrogate model formulates a graph of hills and valleysrepresentative of a location of a tracked object in each frame of thevideo.

Example 17 includes the semiconductor package apparatus of example 11,wherein the Siamese neural network is trained as a general objectsimilarity function.

Example 18 includes the semiconductor package apparatus of example 11,wherein the search space comprises a d×d grid, and wherein the surrogatemodel is rendered over the d×d grid.

Example 19 includes the semiconductor package apparatus of example 11,wherein logic to continue comparison of the new candidate crop to theexemplar crop using the Siamese neural network until the exemplar crophas been found in the current frame further includes logic coupled tothe substrate to compare N new candidate crops to the exemplar cropusing the Siamese neural network and select the new candidate crop withthe highest similarity score as the exemplar crop in the frame.

Example 20 includes the semiconductor package apparatus of example 11,wherein logic to continue comparison of the new candidate crop to theexemplar crop using the Siamese neural network until the exemplar crophas been found in the current frame further includes logic coupled tothe substrate to continue comparison of the new candidate crop to theexemplar crop using the Siamese neural network until a confidence levelin the similarity score of a current new candidate crop is sufficientlyhigh to indicate that the exemplar crop has been found.

Example 21 includes a method of video tracking comprising selecting anexemplar crop to be tracked in an initial frame of a video, applyingBayesian optimization with each subsequent frame of the video bybuilding a surrogate model of an objective function using GaussianProcess Regression (GPR) based on similarity scores of candidate cropscollected from a search space in a current frame of the video, anddetermining where to proficiently obtain a next candidate crop in thesearch space using an acquisition function, comparing the next candidatecrop to the exemplar crop using a Siamese neural network, and continuingcomparison of a new candidate crop to the exemplar crop using theSiamese neural network until the exemplar crop has been found in thecurrent frame, wherein the new candidate crop is selected based on anupdated surrogate model.

Example 22 includes the method of example 21, wherein building asurrogate model of the objective function comprises obtaining candidatecrops of an image in the search space of the current frame of the video,obtaining a similarity score for each of the candidate crops by passingeach of the candidate crops through the Siamese neural network, anddrawing a graph over the image using each similarity score from each ofthe candidate crops to interpolate similarity scores of locations in theframe that have not yet been sampled.

Example 23 includes the method of example 21, wherein determining whereto proficiently obtain the next candidate crop in the search space usingthe acquisition function comprises applying the acquisition function toeach of the candidate crops in the search space to obtain anapproximation of a similarity score for the candidate crops as if eachof the candidate crops had been passed through a Siamese network withoutpassing the candidate crops through the Siamese network, and selecting acandidate crop from the search space that has a maximum acquisitionfunction result as the next candidate crop.

Example 24 includes the method of example 21, wherein the acquisitionfunction comprises a memory-score expected-improvement (MS-EI)acquisition function.

Example 25 includes the method of example 21, wherein comparing the nextcandidate crop to the exemplar object comprises passing the nextcandidate crop through the Siamese network to obtain a similarity score,updating the surrogate model with the new similarity score, and findinga maximum point on the updated surrogate model to use as the newcandidate crop.

Example 26 includes the method of example 21, wherein the surrogatemodel formulates a graph of hills and valleys representative of alocation of a tracked object in each frame of the video.

Example 27 includes the method of example 21, wherein the Siamese neuralnetwork is trained as a general object similarity function.

Example 28 includes the method of example 21, wherein the search spacecomprises a d×d grid, wherein the surrogate model is rendered over thed×d grid.

Example 29 includes the method of example 21, wherein continuingcomparison of the new candidate crop to the exemplar crop using theSiamese neural network until the exemplar crop has been found in thecurrent frame comprises comparing N new candidate crops to the exemplarcrop using the Siamese neural network and selecting the new candidatecrop with the highest similarity score as the exemplar crop in theframe.

Example 30 includes the method of example 21, wherein continuingcomparison of the new candidate crop to the exemplar crop using theSiamese neural network until the exemplar crop has been found in thecurrent frame comprises continuing comparison of the new candidate cropto the exemplar crop using the Siamese neural network until a confidencelevel in the similarity score of a current new candidate crop issufficiently high to indicate that the exemplar crop has been found.

Example 31 includes a computer system for video tracking comprising oneor more processors, one or more memory devices coupled to the one ormore processors, the one or more memory devices including instructions,which when executed by the one or more processors, cause the computersystem to select an exemplar crop to be tracked in an initial frame of avideo, apply Bayesian optimization with each subsequent frame of thevideo to build a surrogate model of an objective function using GaussianProcess Regression (GPR) based on similarity scores of candidate cropscollected from a search space in a current frame of the video, anddetermine where to proficiently obtain a next candidate crop in thesearch space using an acquisition function, compare the next candidatecrop to the exemplar crop using a Siamese neural network, and continuecomparison of a new candidate crop to the exemplar crop using theSiamese neural network until the exemplar crop has been found in thecurrent frame, wherein the new candidate crop is selected based on anupdated surrogate model.

Example 32 includes the computer system of example 31, whereininstructions to build a surrogate model of the objective functionfurther comprises instructions, which when executed by the one or moreprocessors, cause the computer system to obtain candidate crops of animage in the search space of the current frame of the video, obtain asimilarity score for each of the candidate crops by passing each of thecandidate crops through the Siamese neural network, and draw a graphover the image using each similarity score from each of the candidatecrops to interpolate similarity scores of locations in the frame thathave not yet been sampled.

Example 33 includes the computer system of example 31, whereininstructions to determine where to proficiently obtain the nextcandidate crop in the search space using the acquisition functionfurther comprises instructions, which when executed by the one or moreprocessors, cause the computer system to apply the acquisition functionto each of the candidate crops in the search space to obtain anapproximation of a similarity score for the candidate crops as if eachof the candidate crops had been passed through a Siamese network withoutpassing the candidate crops through the Siamese network, and select acandidate crop from the search space that has a maximum acquisitionfunction result as the next candidate crop.

Example 34 includes the computer system of example 31, wherein theacquisition function comprises a memory-score expected-improvement(MS-EI) acquisition function.

Example 35 includes the computer system of example 31, whereininstructions to compare the next candidate crop to the exemplar objectfurther comprises instructions, which when executed by the one or moreprocessors, cause the computing system to pass the next candidate cropthrough the Siamese network to obtain a similarity score, update thesurrogate model with the new similarity score, and find a maximum pointon the updated surrogate model to use as the new candidate crop.

Example 36 includes the computer system of example 31, wherein thesurrogate model formulates a graph of hills and valleys representativeof a location of a tracked object in each frame of the video.

Example 37 includes the computer system of example 31, wherein theSiamese neural network is trained as a general object similarityfunction.

Example 38 includes the computer system of example 31, wherein thesearch space comprises a d×d grid, and wherein the surrogate model isrendered over the d×d grid.

Example 39 includes the computer system of example 31, whereininstructions to continue comparison of the new candidate crop to theexemplar crop using the Siamese neural network until the exemplar crophas been found in the current frame further comprises instructions,which when executed by the one or more processors, cause the computersystem to compare N new candidate crops to the exemplar crop using theSiamese neural network and select the new candidate crop with thehighest similarity score as the exemplar crop in the frame.

Example 40 includes the computer system of example 31, whereininstructions to continue comparison of the new candidate crop to theexemplar crop using the Siamese neural network until the exemplar crophas been found in the current frame further comprises instructions,which when executed by the one or more processors, cause the computersystem to continue comparison of the new candidate crop to the exemplarcrop using the Siamese neural network until a confidence level in thesimilarity score of a current new candidate crop is sufficiently high toindicate that the exemplar crop has been found.

Example 41 includes an apparatus for video tracking comprising means forselecting an exemplar crop to be tracked in an initial frame of a video,means for applying Bayesian optimization with each subsequent frame ofthe video by means for building a surrogate model of an objectivefunction using Gaussian Process Regression (GPR) based on similarityscores of candidate crops collected from a search space in a currentframe of the video, and means for determining where to proficientlyobtain a next candidate crop in the search space using an acquisitionfunction, means for comparing the next candidate crop to the exemplarcrop using a Siamese neural network, and means for continuing comparisonof a new candidate crop to the exemplar crop using the Siamese neuralnetwork until the exemplar crop has been found in the current frame,wherein the new candidate crop is selected based on an updated surrogatemodel.

Example 42 includes the apparatus of example 41, wherein means forbuilding a surrogate model of the objective function comprises means forobtaining candidate crops of an image in the search space of the currentframe of the video, means for obtaining a similarity score for each ofthe candidate crops by passing each of the candidate crops through theSiamese neural network, and means for drawing a graph over the imageusing each similarity score from each of the candidate crops tointerpolate similarity scores of locations in the frame that have notyet been sampled.

Example 43 includes the apparatus of example 41, wherein means fordetermining where to proficiently obtain the next candidate crop in thesearch space using the acquisition function comprises means for applyingthe acquisition function to each of the candidate crops in the searchspace to obtain an approximation of a similarity score for the candidatecrops as if each of the candidate crops had been passed through aSiamese network without passing the candidate crops through the Siamesenetwork, and means for selecting a candidate crop from the search spacethat has a maximum acquisition function result as the next candidatecrop.

Example 44 includes the apparatus of example 41, wherein the acquisitionfunction comprises a memory-score expected-improvement (MS-EI)acquisition function.

Example 45 includes the apparatus of example 41, wherein means forcomparing the next candidate crop to the exemplar object comprises meansfor passing the next candidate crop through the Siamese network toobtain a similarity score, means for updating the surrogate model withthe new similarity score, and means for finding a maximum point on theupdated surrogate model to use as the new candidate crop.

Example 46 includes the apparatus of example 41, wherein the surrogatemodel formulates a graph of hills and valleys representative of alocation of a tracked object in each frame of the video.

Example 47 includes the apparatus of example 41, wherein the Siameseneural network is trained as a general object similarity function.

Example 48 includes the apparatus of example 41, wherein the searchspace comprises a d×d grid, wherein the surrogate model is rendered overthe d×d grid.

Example 49 includes the apparatus of example 41, wherein means forcontinuing comparison of the new candidate crop to the exemplar cropusing the Siamese neural network until the exemplar crop has been foundin the current frame further comprises means for comparing N newcandidate crops to the exemplar crop using the Siamese neural networkand selecting the new candidate crop with the highest similarity scoreas the exemplar crop in the frame.

Example 50 includes the apparatus of example 41, wherein means forcontinuing comparison of the new candidate crop to the exemplar cropusing the Siamese neural network until the exemplar crop has been foundin the current frame further comprises means for continuing comparisonof the new candidate crop to the exemplar crop using the Siamese neuralnetwork until a confidence level in the similarity score of a currentnew candidate crop is sufficiently high to indicate that the exemplarcrop has been found.

Example 51 includes at least one computer readable medium comprising aset of instructions, which when executed by a computing system, causethe computing system to perform the method of any one of examples 21 to30.

Example 52 includes an apparatus comprising means for performing themethod of any one of examples 21 to 30.

Embodiments are applicable for use with all types of semiconductorintegrated circuit (“IC”) chips. Examples of these IC chips include butare not limited to processors, controllers, chipset components,programmable logic arrays (PLAs), memory chips, network chips, systemson chip (SoCs), SSD/NAND controller ASICs, and the like. In addition, insome of the drawings, signal conductor lines are represented with lines.Some may be different, to indicate more constituent signal paths, have anumber label, to indicate a number of constituent signal paths, and/orhave arrows at one or more ends, to indicate primary information flowdirection. This, however, should not be construed in a limiting manner.Rather, such added detail may be used in connection with one or moreexemplary embodiments to facilitate easier understanding of a circuit.Any represented signal lines, whether or not having additionalinformation, may actually comprise one or more signals that may travelin multiple directions and may be implemented with any suitable type ofsignal scheme, e.g., digital or analog lines implemented withdifferential pairs, optical fiber lines, and/or single-ended lines.

Example sizes/models/values/ranges may have been given, althoughembodiments are not limited to the same. As manufacturing techniques(e.g., photolithography) mature over time, it is expected that devicesof smaller size could be manufactured. In addition, well knownpower/ground connections to IC chips and other components may or may notbe shown within the figures, for simplicity of illustration anddiscussion, and so as not to obscure certain aspects of the embodiments.Further, arrangements may be shown in block diagram form in order toavoid obscuring embodiments, and also in view of the fact that specificswith respect to implementation of such block diagram arrangements arehighly dependent upon the computing system within which the embodimentis to be implemented, i.e., such specifics should be well within purviewof one skilled in the art. Where specific details (e.g., circuits) areset forth in order to describe example embodiments, it should beapparent to one skilled in the art that embodiments can be practicedwithout, or with variation of, these specific details. The descriptionis thus to be regarded as illustrative instead of limiting.

The term “coupled” may be used herein to refer to any type ofrelationship, direct or indirect, between the components in question,and may apply to electrical, mechanical, fluid, optical,electromagnetic, electromechanical or other connections. In addition,the terms “first”, “second”, etc. may be used herein only to facilitatediscussion, and carry no particular temporal or chronologicalsignificance unless otherwise indicated.

As used in this application and in the claims, a list of items joined bythe term “one or more of” may mean any combination of the listed terms.For example, the phrases “one or more of A, B or C” may mean A; B; C; Aand B; A and C; B and C; or A, B and C.

Those skilled in the art will appreciate from the foregoing descriptionthat the broad techniques of the embodiments can be implemented in avariety of forms. Therefore, while the embodiments have been describedin connection with particular examples thereof, the true scope of theembodiments should not be so limited since other modifications willbecome apparent to the skilled practitioner upon a study of thedrawings, specification, and following claims.

What is claimed is:
 1. At least one computer readable medium, comprisinga set of instructions, which when executed by one or more computingdevices, cause the one or more computing devices to: select an exemplarcrop to be tracked in an initial frame of a video; apply Bayesianoptimization with each subsequent frame of the video to: build asurrogate model of an objective function using Gaussian ProcessRegression (GPR) based on similarity scores of candidate crops collectedfrom a search space in a current frame of the video; and determine whereto proficiently obtain a next candidate crop in the search space usingan acquisition function; compare the next candidate crop to the exemplarcrop using a Siamese neural network; and continue comparison of a newcandidate crop to the exemplar crop using the Siamese neural networkuntil the exemplar crop has been found in the current frame, wherein thenew candidate crop is selected based on an updated surrogate model. 2.The at least one computer readable medium of claim 1, whereininstructions to build a surrogate model of the objective functionfurther comprises instructions, which when executed by the one or morecomputing devices, cause the one or more computing devices to: obtaincandidate crops of an image in the search space of the current frame ofthe video; obtain a similarity score for each of the candidate crops bypassing each of the candidate crops through the Siamese neural network;and draw a graph over the image using each similarity score from each ofthe candidate crops to interpolate similarity scores of locations in theframe that have not yet been sampled.
 3. The at least one computerreadable medium of claim 1, wherein instructions to determine where toproficiently obtain the next candidate crop in the search space usingthe acquisition function further comprises instructions, which whenexecuted by the one or more computing devices, cause the one or morecomputing devices to: apply the acquisition function to each of thecandidate crops in the search space to obtain an approximation of asimilarity score for the candidate crops as if each of the candidatecrops had been passed through a Siamese network without passing thecandidate crops through the Siamese network; and select a candidate cropfrom the search space that has a maximum acquisition function result asthe next candidate crop.
 4. The at least one computer readable medium ofclaim 1, wherein the acquisition function comprises a memory-scoreexpected-improvement (MS-EI) acquisition function.
 5. The at least onecomputer readable medium of claim 1, wherein instructions to compare thenext candidate crop to the exemplar object further comprisesinstructions, which when executed by the one or more computing devices,cause the one or more computing devices to: pass the next candidate cropthrough the Siamese network to obtain a similarity score; update thesurrogate model with the new similarity score; and find a maximum pointon the updated surrogate model to use as the new candidate crop.
 6. Theat least one computer readable medium of claim 1, wherein the surrogatemodel formulates a graph of hills and valleys representative of alocation of a tracked object in each frame of the video.
 7. The at leastone computer readable medium of claim 1, wherein the Siamese neuralnetwork is trained as a general object similarity function.
 8. The atleast one computer readable medium of claim 1, wherein the search spacecomprises a d×d grid, and wherein the surrogate model is rendered overthe d×d grid.
 9. The at least one computer readable medium of claim 1,wherein instructions to continue comparison of the new candidate crop tothe exemplar crop using the Siamese neural network until the exemplarcrop has been found in the current frame further comprises instructions,which when executed by the one or more computing devices, cause the oneor more computing devices to compare N new candidate crops to theexemplar crop using the Siamese neural network and select the newcandidate crop with the highest similarity score as the exemplar crop inthe frame.
 10. The at least one computer readable medium of claim 1,wherein instructions to continue comparison of the new candidate crop tothe exemplar crop using the Siamese neural network until the exemplarcrop has been found in the current frame further comprises instructions,which when executed by the one or more computing devices, cause the oneor more computing devices to continue comparison of the new candidatecrop to the exemplar crop using the Siamese neural network until aconfidence level in the similarity score of a current new candidate cropis sufficiently high to indicate that the exemplar crop has been found.11. A semiconductor package apparatus comprising: one or moresubstrates; and logic coupled to the one or more substrates, wherein thelogic includes one or more of configurable logic or fixed-functionalityhardware logic, the logic coupled to the one or more substrates to:select an exemplar crop to be tracked in an initial frame of a video;apply Bayesian optimization with each subsequent frame of the video to:build a surrogate model of an objective function using Gaussian ProcessRegression (GPR) based on similarity scores of candidate crops collectedfrom a search space in a current frame of the video; and determine whereto proficiently obtain a next candidate crop in the search space usingan acquisition function; compare the next candidate crop to the exemplarcrop using a Siamese neural network; and continue comparison of a newcandidate crop to the exemplar crop using the Siamese neural networkuntil the exemplar crop has been found in the current frame, wherein thenew candidate crop is selected based on an updated surrogate model. 12.The semiconductor package apparatus of claim 11, wherein logic to builda surrogate model of the objective function further includes logiccoupled to the substrate to: obtain candidate crops of an image in thesearch space of the current frame of the video; obtain a similarityscore for each of the candidate crops by passing each of the candidatecrops through the Siamese neural network; and draw a graph over theimage using each similarity score from each of the candidate crops tointerpolate similarity scores of locations in the frame that have notyet been sampled.
 13. The semiconductor package apparatus of claim 11,wherein logic to determine where to proficiently obtain the nextcandidate crop in the search space using the acquisition functionfurther includes logic coupled to the substrate to: apply theacquisition function to each of the candidate crops in the search spaceto obtain an approximation of a similarity score for the candidate cropsas if each of the candidate crops had been passed through a Siamesenetwork without passing the candidate crops through the Siamese network;and select a candidate crop from the search space that has a maximumacquisition function result as the next candidate crop.
 14. Thesemiconductor package apparatus of claim 11, wherein the acquisitionfunction comprises a memory-score expected-improvement (MS-EI)acquisition function.
 15. The semiconductor package apparatus of claim11, wherein logic to compare the next candidate crop to the exemplarobject further includes logic coupled to the substrate to: pass the nextcandidate crop through the Siamese network to obtain a similarity score;update the surrogate model with the new similarity score; and find amaximum point on the updated surrogate model to use as the new candidatecrop.
 16. The semiconductor package apparatus of claim 11, wherein thesurrogate model formulates a graph of hills and valleys representativeof a location of a tracked object in each frame of the video.
 17. Thesemiconductor package apparatus of claim 11, wherein the Siamese neuralnetwork is trained as a general object similarity function.
 18. Thesemiconductor package apparatus of claim 11, wherein the search spacecomprises a d×d grid, and wherein the surrogate model is rendered overthe d×d grid.
 19. A method of video tracking comprising: selecting anexemplar crop to be tracked in an initial frame of a video; applyingBayesian optimization with each subsequent frame of the video by:building a surrogate model of an objective function using GaussianProcess Regression (GPR) based on similarity scores of candidate cropscollected from a search space in a current frame of the video; anddetermining where to proficiently obtain a next candidate crop in thesearch space using an acquisition function; comparing the next candidatecrop to the exemplar crop using a Siamese neural network; and continuingcomparison of a new candidate crop to the exemplar crop using theSiamese neural network until the exemplar crop has been found in thecurrent frame, wherein the new candidate crop is selected based on anupdated surrogate model.
 20. The method of claim 19, wherein building asurrogate model of the objective function comprises: obtaining candidatecrops of an image in the search space of the current frame of the video;obtaining a similarity score for each of the candidate crops by passingeach of the candidate crops through the Siamese neural network; anddrawing a graph over the image using each similarity score from each ofthe candidate crops to interpolate similarity scores of locations in theframe that have not yet been sampled.
 21. The method of claim 19,wherein determining where to proficiently obtain the next candidate cropin the search space using the acquisition function comprises: applyingthe acquisition function to each of the candidate crops in the searchspace to obtain an approximation of a similarity score for the candidatecrops as if each of the candidate crops had been passed through aSiamese network without passing the candidate crops through the Siamesenetwork; and selecting a candidate crop from the search space that has amaximum acquisition function result as the next candidate crop.
 22. Themethod of claim 19, wherein the acquisition function comprises amemory-score expected-improvement (MS-EI) acquisition function.
 23. Themethod of claim 19, wherein comparing the next candidate crop to theexemplar object comprises: passing the next candidate crop through theSiamese network to obtain a similarity score; updating the surrogatemodel with the new similarity score; and finding a maximum point on theupdated surrogate model to use as the new candidate crop.
 24. The methodof claim 19, wherein the surrogate model formulates a graph of hills andvalleys representative of a location of a tracked object in each frameof the video.
 25. The method of claim 19, wherein the Siamese neuralnetwork is trained as a general object similarity function.